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Tutorial  Composition 


•  Basic  Concepts 

•  Break 

•  Advanced  Concepts 

•  Break 

•  Detailed  Examples 

•  Wrap-Up 


Design  of  Experiments 


Basic  Concepts 
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Summary  Slide 


•  What  is  DoE? 

•  Purposes  of  Experimenting 

•  Experimentation  Strategies 

•  Basic  Principles 

•  Nuisance  Factors 

•  Design  Steps 

•  Major  Guidelines 

•  Simple  Comparison  Experiments 

•  Single  Factor  Experiments 

•  Latin  Squares 
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What  is  DoE? 


Experiment:  a  test  or  series  of  tests  where 
the  experimenter  makes  purposeful  changes 
to  input  variables  of  a  process  or  system  so 
that  we  can  observe  or  identify  the  reasons 
for  changes  in  the  output  responses. 


•  Design  of  Experiments:  is  concerned  with 
the  planning  and  conduct  of  experiments  to 
analyze  the  resulting  data  so  that  we  obtain 
valid  and  obiective  conclusions. 


Lineage 
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1 771  -  Course  of  Experimental  Agriculture 
Arthur  Young 

-  One  of  the  earliest  direct  experimental  scientific  documents 

-  Insisted  on  split-field  trials 

-  Required  repeated  trials  in  different  fields 

1919  -  R.A.  Fisher  started  work  as  a 
statistician  at  Rothamsted  Agricultural 
Experimental  Station 

-  Randomization  of  trials 

-  Creation  of  the  technique  “Analysis  of  Variance” 

Today.... 


Code  of  Best  Practices 
(COBP) 
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Code  of  Best  Practice  for  Experimentation, 
CCRP,  2002 

Campaigns  of  Experimentation:  Pathways  to 
Innovation  and  Transformation,  Alberts  & 
Hayes,  2005 

These  documents  identify  3  types  of 
experiments: 

-  Discovery 

-  Hypothesis 

-  Demonstration 

This  tutorial  focuses  on  aspects  of  the  first 
two  types 
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Types  of  Experiments 


•  Discovery 

-  Designed  to  generate  new  ideas  or  approaches 

-  Usually  involve  “hands-on”  activities 

-  May  involve  systems  or  processes  that  are  not  well 
understood  or  refined 

•  Hypothesis 

-  Closer  to  the  traditional  academic  approach 

-  Seek  to  falsify  specific  hypotheses 

-  Used  often  in  the  attempt  to  “prove”  a  theory,  idea, 
approach 


Why  Experiment? 


Determine  which  variables  are  the  most 
influential  in  a  process  or  system 

Determine  where  to  set  the  inputs  so  the 
output  is  always  near  the  desired  state 

Determine  where  to  set  the  inputs  so  the 
output  variability  is  minimized 

Determine  where  to  set  the  inputs  so  the 
influence  of  uncontrollable  factors  is 
minimized  (robust  design) 
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Experimentation  Strategies 


Best  Guess 

-  PRO:  Works  reasonably  well  when  used  by  SMEs  with  solid 
foundational  knowledge  on  known  issues 

-  CONs: 

■  If  it  fails,  need  to  guess  again... and  again... until.... 

■  If  get  acceptable  results  first  time,  may  stop  without  discovering  “better” 

One  Factor  at  a  Time 

-  PRO:  Straight-forward,  easily  understood 

-  CONs: 

■  Impossible  to  address  interactions 

■  Tends  to  “over  collect”  data,  not  efficient  sample  sizes 

Factorial 

-  pros: 

■  Full  evaluation  of  individual  and  interaction  effects 

■  Most  efficient  design  with  respect  to  sample  sizes 

-  CON:  More  complex  to  explain  to  untrained  audiences 
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Basic  Principles 


DdE  Tutorial 


•  Replication 

-  Permits  estimation  of  experimental  error 

-  Permits  more  precise  estimates  of  the  sampie  statistics  ■ 

-  Not  to  be  confused  with  repeated  measures 

•  Randomization 

-  insures  that  observations  or  errors  are  more  likely  to  be 
independent 

-  Helps  “average  out”  effects  of  extraneous  factors 

-  Special  designs  when  compiete  randomization  not  feasible 

•  Blocking 

-  Designed  to  improve  precision  of  comparisons 

-  Used  to  reduce  or  eliminate  nuisance  factors 
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Nuisance  Factors 


Definition:  A  nuisance  factor  is  a  “design 
factor  that  probably  has  an  effect  on  the 
response  but  we  are  not  interested  in  that 

effect”  [Montgomery,  p126,  emphasis  added] 

Nuisance  Factors,  Types  ^  Cures 


-  Known  and  controllable  ^  Use  blocking  to  systematically 
eliminate  the  effect 


-  Known  but  uncontrollable  ^  If  it  can  be  measured,  use 
Analysis  of  Covariance  (ANCOVA) 

-  Unknown  and  uncontrollable  ^  Randomization  is  the 
insurance 


Design  Steps 


•  Recognition  and  statement  of  the  problem  in 
nonstatistical  language 

•  Selection  of  factors,  levels,  ranges 

•  Selection  of  response  variables 

•  Choice  of  experimental  design 

•  Performance  of  the  experiment 

•  Statistical  analysis  of  the  data 

•  Conclusions  and  recommendations 


Major  Guidelines 
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Use  team’s  non-statistical  knowledge  of  the 
problem  to: 

-  Choose  factors 

-  Determine  proper  levels 

-  Decide  number  of  replications 

-  Interpret  results 

Keep  the  design  and  analysis  as  simple  as 
possible 

Recognize  the  difference  between  practical 
and  statistical  significance 

Be  prepared  to  iterate  -  commit  no  more  than 
25%  of  available  resources  to  first  series 


Simple  Comparison 
Experiments 
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Goal: 

-  Compare  two  or  more  means;  variances;  probabilities 

-  Compare  A  versus  B\  [better  or  worse]  -  paired  comparison 
is  a  special  case  of  randomized  block  design 

Major  Considerations 

-  Sample  size 

-  Distributional  knowledge:  Normal,  F . etc. 

-  Structure  of  the  statistical  hypothesis 

■  One-tailed  Two-tailed  tests 
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Single  Factor  Experiments 


DoB  Tutorial 


•  One  Factor  -  Multiple  Levels  ■ 

•  “One-level-at-a-time”  analysis  isn’t  efficient 

-  Consider  one  factor  with  five  levels 

-  Pair-wise  comparison  requires  10  pairs  [  5C2  =  10  ] 

-  If  each  comparison  has  a  =  0.05,  then 

Probability(correct  assessment)  =  (1-a)''°  =  0.60 

•  Technique  of  Choice  -  ANOVA 

-  Tests  hypothesis  Hg:  p.,  =  ILI2  =  ILI3  =  ... 

-  Assumptions 

■  Error  term  is  Normal  (O.a^)  ^  test  residuals  to  confirm 

■  Conditions  properly  randomized 

■  Results  are  independent;  errors  are  independent 

-  If  reject  Hq  (i.e.,  failed  the  test)  then  use  Newman-Keuls  Range  Test 
or  Duncan’s  Multiple  Range  Test  to  determine  the  specifics 

-  Note  -  there  are  non-parametric  tests  in  lieu  of  ANOVA  if 
assumptions  are  not  met  (e.g.  Kruskal-Wallis  Test) 
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Latin  Squares 


Latin  Square:  An  arrangement  of  conditions 
such  that  each  combination  occurs  only  once 
in  each  row  and  column  of  the  test  matrix. 
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B 
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Graeco-Latin  Square:  The  superposition  of  two  Latin 
Squares  such  that  each  paired-combination  occurs 
only  once  in  each  row  and  column. 
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Orthogonal 

Latin 

Square 
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Latin  Squares  -  Practical 

Example 


Conduct  a  test  of  new  intelligence  fusion  procedures  using  four 
analyst  teams  examining  four  scenarios.  Each  fusion  process 
will  take  one  day  of  test  activity  to  fully  work  the  process. 

-  Day  1  ^  Orientation  Day  for  participants;  assign  teams  (A,B,C,D) 

-  Days  2  through  5  =>  Test  days 


Scenario  1 
Scenario  2 
Scenario  3 
Scenario  4 
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Do  it  again,  3  months  later  with  different  teams  (a,p,x,5) 


Scenario  1 
Scenario  2 
Scenario  3 
Scenario  4 


a 
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5 

p 

X 

8 

a 

X 

5 

a 

p 

6 

a 

p 

X 

Combine  analytical  results 


Scenario  1 
Scenario  2 
Scenario  3 
Scenario  4 


Day  1 

Day  2 

Day  3 

Day  4 

A  a 

Bp 

C  X 

D6 

Bp 

C  X 

D6 

A  a 

C  X 

D6 

A  a 

Bp 

D8 

A  a 

Bp 

_ _ 

Non- 

Orthogonal 

Latin 

Square 
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Advanced  Concepts 


Lockheed  Martin 
Center  for  Innovation 


Segment  Agenda 


•Advanced  DoE 

-  Factorials 

■  Full 

■  Fractional 

■  Other  Types 

-  Complex  Designs 


Full  Factorial  Designs 
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Definition:  An  experiment  in  which  for  each 
completed  trial  or  replication  of  the 
experiment  all  possible  combinations  of  the 
levels  of  the  factors  are  investigated. 

Design  Notation 

-  General  Notation  for  2-level  experiment  ^  2^  where  k  = 
number  of  factors 

■  3  factors  2  levels  each  =  2^  design 

-  Factors  and  Levels  ^  example  for  3  factors,  2  levels 

■  Aa  Bb  Cc 

■  A^A-B^B-CX- 

■  (1)abc 
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Full  Factorial  Design 


All  combinations  are  examined 

-  Example  2^  design  =  8  experimental  settings: 

A-B+C+  A^B-C*  A+B^C’  A-B'C^  A-B+C'  A+B'C’  A  B  C' 

Effects  Evaluated 

-  Main  effects  of  single  factors:  A,  B,  C 

-  Second  Order  (2-factor)  interactions:  AB,  AC,  BC 

-  Third  Order  (3-factor)  interactions:  ABC 

-  In  general,  a  2*^  design  evaluates  all  1, 2,  ...k-1,  k-factor 
effects 


•  Advantages  over  “one-factor-at-a-time” 

-  More  efficient  in  time,  resources,  sample  size 

-  Addresses  interactions 

-  Provides  insight  over  a  range  of  experimental  conditions 
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Factorial  Efficiency  - 
Graphically  (1) 
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Factorial  Efficiency  - 
Graphically  (2) 


Main  Effect  A 

=  (1/4nj  *  [blue  square 
-red  square] 
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Factorial  Efficiency  - 
Graphically  (3) 


Main  Effect  B 

=  (1/4nj  *  [blue  square 
-red  square] 


be 


abc 
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Factorial  Efficiency  - 
Graphically  (4) 


Main  Effect  C 

=  (1/4nj  *  [blue  square 
-red  square] 


be 


abc 
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Factorial  Efficiency  - 
Graphically  (5) 


Effect  AB 

=  {M4n)  *  [blue  plane 
-red  plane] 


be 


abc 
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Factorial  Efficiency  - 
Graphically  (6) 


Effect  AC 

=  {MAn)  *  [blue  plane 
-red  plane] 


be 


Lockheed  Martin 
Center  for  Innovation 


Factorial  Efficiency  - 
Graphically  (7) 


Effect  BC 

=  {M4n)  *  [blue  plane 
-red  plane] 


be 


abc 
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Factorial  Statistics  (1) 


Standard  ANOVA  table 


Factors 


ANOVA  for  Selected  Factorial  Model 

Analysis  of  variance  table  [Partial  sum  of  squares] 

Sum  of 

Mean 

F 

Source 

Squares 

DF 

Square 

Value 

Prob  >  F 

Model 

52202222.22 

7 

7457460.32 

182.57 

<  0.0001  significant 

A 

16885313.28 

1 

16885313.28 

413.38 

<  0.0001 

C 

30040938.28 

1 

30040938.28 

735.45 

<  0.0001 

E 

51280.03 

1 

51280.03 

1.26 

0.2736 

AC 

15268.78 

1 

15268.78 

0.37 

0.5467 

AE 

2896222.78 

1 

2896222.78 

70.90 

<  0.0001 

CE 

533286.28 

1 

533286.28 

13.06 

0.0014 

ACE 

1779912.78 

1 

1779912.78 

43.58 

<  0.0001 

Residual 

980327.75 

24 

40846.99 

Cor  Total 

53182549.97 

31 

Significance 

Level 


Factorial  Statistics  (2) 


Additional  Statistics 


Standard  deviation 
of  the  main 
measure 


Measures  the 
proportion  of  total 
variability  explained 
by  the  model 


Std.  Dev. 


Mean 


C.V. 


PRESS 


202.1063818 


8439.46875 


2.394776115 


1742804.889 


R-Squared 

0.98156674 

AdJ  R-Squared 

0.97619037 

Pred  R-Squared 

0.96722976 

Adeq  Precision 

_ 

42.0620068 

Mean  of  the  main 
measure 


PRESS  =  Prediction  Error  Sum 
of  Squares 

•  A  measure  of  how  well  the 
model  will  “predict”  new  data 

•  Smaller  is  better  but  can  only 
be  used  in  a  comparative  sense 


•  R2  adjusted  for  the  number 
of  Factors 

•  If  non-significant  terms  are 
“forced”  into  the  model  this 
can  decrease 


An  estimate  of  the 
amount  of  variability 
in  the  new  data  that 
would  be  explained 
by  the  full  model 


Measures  the  signal-to-noise  ratio  in  the  data 
•  An  indicator  if  Response  Surface  Methods 
(RSM)  are  applicable 
Values  >4  are  good 
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Link  to  Linear  Regression 


•  Final  Model  looks  like  a  regression  equation 


MoM 

= 

8439.46875 

726.40625 

*  A 

-968.90625 

*C 

40.03125 

*  E 

-21.84375 

*A*C 

300.84375 

*  A*  E 

-129.09375 

*  C  *  E 

235.84375 

*A*C*E 

•  Tests  of  Significance 

-  Overall  model  response 

-  Individual  coefficients 

•  Diagnostic  tests 

-  Residuals 

-  Outliers 

-  Lack  of  Fit 


•  A  way  to  reduce  a  huge  full  factorial  to  something  | 
manageable 

-  Considerations  I 

■  Required  time,  resources 

■  Complexity  of  set-up  for  experiments 

-  Major  use  is  in  screening  experiments  where  the  knowledge  of  | 
basic  effects  is  not  well  known 

-  If  2'^  is  very  large,  may  need  to  run  reduced  experiment  ^ 

•  Justification 

-  Sparsity  of  Effects  -  in  general,  even  complex  systems  are  usually 
driven  by  a  few  main  effects  and  low-level  interactions 

-  Projection  Property  -  fractional  factorial  designs  can  be  “projected” 
into  larger  designs  in  the  subset  of  significant  factors 

-  Sequential  Experimentation  -  can  combine  runs  of  2  or  more 
fractional  designs  into  larger  designs 
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•  Issue: 

-  Confounding  of  Effects  (also  called  “aliasing”)  ^  reduced 
experiments  do  not  evaluate  all  levels  of  the  factors  and  their 
interactions 

-  Some  mixture  of  effects  is  “confounded”  and  not  identifiable 

•  Challenge: 

-  To  select  the  best  combination  of  test  elements  that  stands  a 
reasonable  chance  of  revealing  the  true  effects 

-  Alias  the  (most  likely)  insignificant  or  unwanted  factors 

•  Symbology 

-  2'^-P  designs 


•  Resolution  ^  a  measure  of  confounding 


-  Resolution  III 


■  No  main  effect  aliased  with  any  other  main  effect 

■  Main  effects  are  aliased  with  2-factor  interactions 

■  2-factor  interactions  may  be  aliased  with  each  other 

-  Resolution  IV 


jk-p 


■  No  main  effect  aliased  with  any  other  main  effect 

■  No  main  effect  aliased  with  2-factor  interactions 

■  2-factor  interactions  may  be  aliased  with  each  other 

-  Resolution  V 


ryk-p 

^IV 


■  No  main  effect  aliased  with  any  other  main  effect 

■  No  main  effect  aliased  with  2-factor  interactions 

■  No  2-factor  interactions  may  be  aliased  with  each  other 

■  2-factor  interactions  are  aliased  with  3-factor  interactions 


ryk-p 


Experiments 
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Resolution  Trade-offs 
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Number  of  Factors 


12 


13 


14 


1/2 

Fract. 


1/4 

Fract. 


1/16 

Fract. 


1/8 

Fract. 


1/16 

Fract. 


1/32 

Fract. 


1/64 

Fract. 


1/128 

Fract. 


1/256 

Fract. 


1/512 

Fract. 


15 


1/1024 

Fract. 


Full 


1/2 

Fract. 


1/4 

Fract. 


1/8 

Fract. 


1/16 

Fract. 


1/32 

Fract. 


1/64 

Fract. 


1/128 

Fract. 


1/256 

Fract. 


1/512 

Fract. 


Full 


1/2 

Fract. 


1/4 

Fract. 


1/8 

Fract. 


1/16 

Fract. 


1/32 

Fract. 


1/64 

Fract. 


1/128 

Fract. 


1/256 

Fract. 


256 


Full 


1/2 

Fract. 


1/4 

Fract. 


1/8 

Fract. 


1/16 

Fract. 


1/32 

Fract. 


1/64 

Fract. 


1/128 

Fract. 


•  Green  =  Resolution  V 


•  Yellow  =  Resolution  IV 

•  Red  =  Resolution  III 


Lockheed  Martin 
Center  for  Innovation 


Half  Replicate/Folding 


STD 

RUN 

A 

B 

C 

D 

E 

1  1 

7 

-1 

-1 

-1 

-1 

1 

2 

4 

1 

-1 

-1 

-1 

-1 

3 

6 

-1 

1 

-1 

-1 

-1 

4 

10 

1 

1 

-1 

-1 

1 

5 

3 

-1 

-1 

1 

-1 

-1 

6 

14 

1 

-1 

1 

-1 

1 

7 

5 

-1 

1 

1 

-1 

1 

8 

1 

1 

1 

1 

-1 

-1 

9 

8 

-1 

-1 

-1 

1 

-1 

10 

2 

1 

-1 

-1 

1 

1 

11 

11 

-1 

1 

-1 

1 

1 

12 

15 

1 

1 

-1 

1 

-1 

13 

9 

-1 

-1 

1 

1 

1 

14 

13 

1 

-1 

1 

1 

-1 

15 

12 

-1 

1 

1 

1 

-1 

16 

16 

1 

1 

1 

1 

1 

other  Design  Variations  (1) 


DoB  Tutorial 


Three  levels  for  k-factors  (S*^)  designs 
Fractional  3-level  designs  (3'^-p) 

Adding  Center  runs  to 

-  Get  estimates  of  process  variability 

-  Gain  familiarity  with  the  process 

-  Identify  system  performance  limits 

Mixture  Designs  -  where  one  or  more  factors 
are  constrained  to  add  to  something 

-  Usually  have  constraints  like:  +  X2  +  X3  +  ...+Xp  =  1 

-  Example:  A  mixture  of  contributing  probabilities 

Nested  and  Split-Plot  designs  for  experiments 
with  random  factors 


other  Design  Variations  (2) 


DdE  Tuturiii! 


•  Irregular  Fraction 

-  Usually  a  Resolution  V  design  for  4  to  9  factors  where  each 
factor  is  varied  over  only  2  levels 

-  Two-factor  interactions  aliased  with  three-factor  and  higher 

-  Excellent  to  reduce  number  of  runs  and  still  get  clean  results 

•  General  Factorial 

-  For  1  to  12  factors  where  each  factor  may  have  a  different 
number  of  levels 

-  All  factors  treated  as  categorical 

•  D-Optimal 

-  A  special  design  for  categorical  factors  based  on  a  analyst- 
specified  model 

-  Design  will  be  a  subset  of  the  possible  combinations 

-  Generated  to  minimize  the  error  associated  with  the  model 
coefficients 


other  Design  Variations  (2) 


DdE  Tuturiii! 


•  Plackett-Burman  ■ 

-  Specialized  design  for  2  to  31  factors  where  each  factor  is 
varied  over  only  2  levels 

-  Use  only  if  you  can  reasonably  assume  NO  two-factor 
interactions;  otherwise,  use  fractional  factorial  designs  J 

•  Taguchi  OA 

-  Saturated  orthogonal  arrays  -  all  main  effects  and  NO  m 
interactions 

-  Special  attention  must  be  paid  to  the  alias  structure  for 
proper  interpretation  at  both  the  design  phase  (prior  to  runs) 
and  during  final  analysis 
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Design  of  Experiments 


Practical  Examples 


Lockheed  Martin 
Center  for  Innovation 


Steps  in  DoE 


Design  the  experiment 
Evaluate  the  design 

-  Model  specification 

-  Power  calculations  (1-p) 

-  Graphical  examination  of  the  standard  error  of  the  design 

Conduct  the  experiment  and  collect  data 
Analyze  the  results 

-  Examine  data  for  transformation  suggestions 

-  Compute  the  effects 

-  Perform  AN  OVA 


-  Critical!!!  -  ALWAYS  check  the  diagnostics 

-  Examine  graphical  findings 

-  Finalize  the  analysis 


.  Lockheed  Martin 
Center  for  Innovation 


Diagnostics 


Diagnostic  steps  are  the  most  often  omitted  -  to  the 
analyst’s  potential  embarrassment 

Which  of  these  ANOVA  tables  are  to  be  believed? 


Intercetat 


Lack  Of  Fit 
Pure  Error 
Residuals 


DF  Sum  of  Squares  [Mean  Square  I  F  Value  |  Prob  >  F  |%  Contribution 

2.75  0.0647 


75.69 

110.03 

0.000 

110.03 


18.92 

6.88 

6.88 


40.76 

59.24 

0.000 


a 

Intercept 

n 

A 

4 

11.68 

2.92 

25.73  <0.0001 

87.28 

e 

Lack  Of  Fit 

15 

1.70 

0.11 

12.72 

e 

Pure  Error 

0 

0.000 

0.000 

Residuals 

15 

1.70 

0.11 

Studentized  Residuals 


Residuals  vs.  Run 


^  Du£  Tuiurivl 


Studentized  Residuals 


Residuals  vs.  Settings 


Tui^riv! 


Residuals  vs.  Settings 


Residuals  vs.  Settings 


Leverage 


Leverage  vs.  Run 


I  '  '  I  '  '  I  '  '  I  '  '  I  '  '  I  '  '  I  '  '  I  '  '  I 

1  4  7  10  13  16  19  22  25 


Tutorial 


Leverage  vs.  Run 


■  ■  ■  ■  ■  ■ 

■ 

■ 

1  4  7  10  13  16  19  22  25 


Run  Number 


Run  Number 


Power  T  ransform 
Recommendation 


DoE  Tutoriv! 


Box-Cox  Plot  for  Power  Transforms 


Lambda 


Lambda 


Cargo  Vehicles 


Single  Factor  Plot 


18.3751  —  ^ 


One  Factor  Plot 


— ^ - 1 — 

LMRSD10  LMRSD15(0L) 


A:  SD 


LMRSD20  LMRSD25 


TuiQrivI 


One  Factor  Plot 


A:  SD 


lijl 


Lockheed  Martin 
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Remember!! 


ALWAYS  perform  the  diagnostic  tests! 
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Measuring  the  Effect  of  C3I  on 

Combat: 

Methodology  and  Results 


An  Example  of  the  Application  of  Design  of 
Experiments  Concepts  and  Techniques 


Initial  Guidance 


>3 


DoB  Tuturivl 


Evaluate  the  impact  of  the 
representations  of  C3I  on  combat 
outcomes  in  a  campaign-level  force-on 
force  model. 

-  Perform  sensitivity  analyses  across  all  areas  of 
C3I,  utilizing  the  existing  test  scenario. 

-  Determine  which  C3l-related  input  data  have  the 
most  impact  on  combat  outcomes. 


Approach 


>3 


DdE  Tutcjrivl 


Select  specific  C3I  functions  to  be  examined 
Design  the  Experiment. 

Prepare  model  software  and  scenario. 
Execute  model  runs. 

Analyze  the  output  and  report  findings. 


Excluded  Elements 


DoB  Tutoriiil 


The  following  items  are  not  part  of  the  study  as 
they  are  either  not  controllable  from  a  military 
sense  or  they  represent  different  tactics  or 
behaviors,  which  are  not  of  interest  for  this 
study: 

>  Weather 

>  Intelligence  Ratings 

>  Force  Structure 

>  ISR  Collection  Plans 


Lockheed  Martin 
Center  for  Innovation 


Experimental  Design 


Bundled  multiple  factors  into  3 
categories  to  describe  C3I  functionality 
in  terms  of: 

-  Timeliness 


-  Quantity 

-  Quality 


Statistical  Considerations 


>3 


DdE  Tuturiii! 


Each  candidate  factor  could  influence 
combat  outcome  either: 

-  By  itself, 

-  In  concert  with  another  factor, 

-  In  opposition  to  another  factor. 

Previous  research  in  this  area  has 
shown  serious  non-linear  effects. 


Formal  Design 


>3 


Three-factor  design  with  interaction  and  non¬ 
linear  terms. 


r 


co  = 


^0  +  +  PiQt  +  A0 


L 


+ 


+ 


+ 


where: 


V. 


A  +  PiiQt  +  AsG 


2 

L 


Pxi^Qt  +  Pn^Qi  +  PizQtQl  +  Pm^QrQ 


L 


+ 


CO=  Combat  Outcome 
T  =  Timeliness 
Qj  =  Quantity 
Ql  =  Quality 

Pi  =  an  unknown  value  to  be  estimated 
s  =  the  error  term 


Lockheed  Martin 
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Measurement  Points 


Face-Centered  Central  Composite  Design 


Design  Matrix 


Tutorial 


The  FC-CCD  yields  the  following  design 
matrix: 


-1  +1  -1  +1  -1  +1  -1  +1 

-1  -1  +1  +1  -1  -1  +1  +1 

-1  -1  -1  -1  +1  +1  +1  +1 

V  ^ 

0  0  0  0  -1  +1 

0  0-1+1  0  0 

-1  +1  0  0  0  0 

V' 

Corner  Points 

V 

Face  Points 

■^T 


SHi 

c 

E 

N 

T 

E 

R 


Parameter  Settings 


DdE  Tutorial 


Each  parameter  was  chosen  so  that  3 
settings  were  possible  to  match  the  FC 
CCD  requirements: 

-  High  (meaning  improved  or  enhanced 
performance) 

-  Center  (baseline) 

-  Low  (meaning  reduced  or  degraded 
performance) 


3^  Lockheed  Martin 
_  JI  Center  for  Innovation 


Timeliness  (T)  Settings 


Parameters  Low(-1)  Center  (0)  High  (+1) 

Reporter  Delay  Time 
(RDT) 

8 

4 

0 

Presented 

Communications  Load 
(PCL) 

1.25*PCLb33, 

PCLb333 

0.75*PCLb333 

Maximum 
Communications 
Network  Capacity 
(MCNC) 

0.75*MCNCb333 

MCNCB3se 

1.25*MCNCB3,e 

3^  Lockheed  Martin 
_  JI  Center  for  Innovation 


Quantity  (Qj)  Settings 


Parameters 

Low  (-1) 

Center  (0) 

High  (+1) 

■MINT  Probability  of 
Detection 

0.4 

0.7 

1.0 

Sensor  Footprint  (SFP) 

0.707*SFPb33, 

SFPB3se 

1.414*SFPb333 

COMINT  Sensor 
Search  Rate  (CSSR) 

0.5*CSSRB33e 

CSSR^ase 

2*CSSRb333 

3^  Lockheed  Martin 
_  JI  Center  for  Innovation 


Quality  (QJ  Settings 


Parameters 

Low  (-1) 

Center  (0) 

High  (+1) 

Probabiiity  of  Correct 
Ciassification  for  MTi 

0.75  :  0.25 

0.5  :  0.5 

0.25  :  0.75 

sensors  (Pcc.mti) 

Quaiity  Probabiiity  for 
expiicit  iMiNT  search 

(^q-imint) 

p 

■^Degrade-Q-IMINT 

P 

■^Q-IMINT 

p 

■^Upgrade-Q-IMINT 

Association  Threshoid 
(AT) 

0.5*ATg3gg 

^^Base 

2*AT 

^  ^ '  Base 

Lockheed  Martin 
Center  for  Innovation 


Probability  Transforms  (1) 


Quality  Classification  Probability  is  actually  a 
distribution,  not  a  single  value,  where  Ip|  =  1 


Quality 

Level 

1 

2 

3 

N 

Probability 

Pi 

P2 

Ps 

Pn 

-<H-|--a3>a3073"D 
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Combat  Outcome  Measures 


Combat  Outcome  Sources 

°K-AII 

Direct  Fire  KVSB* 
Indirect  Fire  KVSB 
Air-to-Ground  KVSB 

°K-DF 

Direct  Fire  KVSB 

Rk-if 

Indirect  Fire  KVSB 

°K-A2G 

Air-to-Ground  KVSB 

*KVSB  =  Killer-Victim  Scoreboard 


Run  Results  Summary 


Response  Name 

Y1  DF  Kills  -  Red 

Y2  IF  Kills  -  Red 

Y3  A2G  Kills  -  Red 

Total  Kills  -  Red 


Observations 

150 

150 

150 

150 


Minimum 

166.696716 

38.507262 

639.270296 


1039.098472  2422.615195 


Maximum 

543.0073751 

317.413308 

1986.671009 


DF  =  Direct  Fire 
IF  =  Indirect  Fire 
A2G  =  Air-to-Ground 


3^  Lockheed  Martin 
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ANOVA- Total  Kills 


Source 

Sum  of 
Squares 

DF 

Mean 

Square 

F 

Value 

Prob  >  F 

a  =  0.05 

Model 

6,024,287.18 

9 

669,365.24 

20.2233 

<  0.0001 

* 

T 

3,235,042.05 

1 

3,235,042.05 

97.739 

<  0.0001 

* 

Qt 

659,344.82 

1 

659,344.82 

19.9205 

<  0.0001 

* 

Ql 

794,562.33 

1 

794,562.33 

24.0058 

<  0.0001 

* 

T2 

212,422.81 

1 

212,422.81 

6.4178 

0.0124 

* 

Qt" 

558,906.79 

1 

558,906.79 

16.886 

<  0.0001 

* 

Ql" 

356,637.37 

1 

356,637.37 

10.7749 

0.0013 

* 

1 - 

P 

17,652.64 

1 

17,652.64 

0.5333 

0.4664 

TQl 

47,410.91 

1 

47,410.91 

1 .4324 

0.2334 

QjQl 

199,200.94 

1 

199,200.94 

6.0184 

0.0154 

* 

Residual 

4,633,827.61 

140 

33,098.77 

Lack  of  Fit 

660,829.65 

5 

132,165.93 

4.4909 

0.0008 

* 

Pure  Error 

3,972,997.96 

135 

29,429.61 

Cor  Total 

10,658,114.79 

149 

■  Lockheed  Martin 
Center  for  Innovation 


Fitted  Model  -  Total  Kills 


co  = 


1595.89  +  179.867  +  81.200^+89.140^  + 

+  90.897"  +147.430^  -117.770" 

- 14.8570,  -  24.3470,  -  49.900,0,  ^ 


Design  Evaluation 


>3 


DdE  Tuturivl 


Evaluate  the  formal  design  by 
examining  the  following  parameters 

-  Calculate  power  of  the  tests 

-  Perturbation  plots 

-  Contour  plots 

-  Standard  error  graphs 


m 


Lockheed  Martin 
m  Center  for  Innovation 


Power  of  the  Design 


Term 

Std  Err** 

Power  at  5%  alpha  level  for  effect  of: 
1/2  Std.  Dev.  1  Std.  Dev.  2  Std.  Dev. 

T 

0.1 

69.90% 

99.90% 

99.90% 

Qt 

0.1 

69.90% 

99.90% 

99.90% 

Ql 

0.1 

69.90% 

99.90% 

99.90% 

0. 1 972027 

71 .20% 

99.90% 

99.90% 

Qt' 

0.1972027 

71.20% 

99.90% 

99.90% 

Ql' 

0.1972027 

71.20% 

99.90% 

99.90% 

1 - 

P 

0.1118034 

60.30% 

99.30% 

99.90% 

TQl 

0.1118034 

60.30% 

99.30% 

99.90% 

QjQl 

0.1118034 

60.30% 

99.30% 

99.90% 

**Basis  Std.  Dev.  =  1.0 

A 

1  T 

'tlTTlI 

>-•  iVIiAl  litl 

Center  for  Innovation 

>  Plot  of  standard 
Error  of  Design 

-  Shows  error  of 
the  estimates 
increases  at  the 
edge  of  the 
design  space 

-  All  factors 
overlap:  they 
have  the  same 
standard  error 

-  Conclusions 
based  on 
extreme  values 
may  be  subject 
to  major 
qualification 


0.184265 


0.167119 


Perturbation 


Deviation  from  Reference  Point 


Lockheed  Martin 
Center  for  Innovation 


Diagnostic  Tests 


•  Examine  data  output  with: 

-  Normal  plot  of  the  residuals 

-  Residuals  vs.  predicted  error 

-  Residuals  vs.  run 

-  Residuals  vs.  Timeliness 

-  Residuals  vs.  Quantity 

-  Residuals  vs.  Quality 

•  Conduct  outlier  investigation 

•  Conduct  transform  analysis 


Lockheed  Martin 
Center  for  Innovation 


>  Outlier  Analysis 

-  Desired  -  strong 
clustering  near 
the  zero  point 

-  Actual  -  strong 
clustering  near 
the  zero  point 

-  Results  -  OK 


Outlier  Investigation  (2) 


DESIGN-EXPERT  Plot 
T otal  Kills  -  Red 


<D 

o 

c 

CD 

Hh-» 

b 

C/) 

Ik 

o 

o 

O 


Cook's  Distance 


1.00- 


0.75- 


0.50- 


0.25- 


1  38  75  112  149 


Run  Number 


Leverage  vs.  Run 


■  ■  ■ 


Lockheed  Martin 
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Predicted  vs.  Actual 


>  Outlier  Analysis 

-  Desired  -  no 
apparent  pattern 
in  the  observed 
data 

-  Actual  -  no 
pattern  in  the 
observed  data 

-  Results  -  OK 


DESIGN-EXPERT  Plot 
T otal  Kills  -  Red 


Predicted  vs.  Actual 


Actual 


Lockheed  Martin 
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Power  Transform 


>  Transform  Analysis 

-  Desired  -  no 
transform 

-  Actual  -  Log 
transform 
recommended 

-  Results - 
transform  not 
pursued  due  to 
constraints: 

■  Time 
available 

■  Quality  of 
data 


DESIGN-EXPERT  Plot 
Total  Kills  -  Red 

Lambda 
Current  =  1 
Best  =  -0.03 
Low  C.l.  =  -0.74 
High  C.l.  =  0.7 

Recommend  transform 
Log 

(Lambda  =  0) 


Box-Cox  Plot  for  Power  Transforms 


Lambda 


Lockheed  Martin 
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Evaluation  Results 


Model  has  statistical  power: 

-  For  Type  I  error  of  5%,  Type  II  error  is  less  than  0.1%. 


•  Diagnostics  acceptable: 

-  No  problems  based  on  residual  analysis. 

-  No  problems  based  on  outlier  analysis. 

-  Data  transform  suggested  but  not  deemed  essential  for  this 


Lockheed  Martin 
Center  for  Innovation 


Interpretation  and  Analysis 


•  Review  the  results  in  terms  of: 

-  ANOVA  Table 

-  Perturbation  plots 

-  Single  factor  response 

-  Interaction  response 

-  Contour  plots 

-  3-D  surface  plots 

-  Cube  plot 


^]l 


Lockheed  Martin 
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AHOVA  -  Total  Kills 


Source 

Sum  of 
Squares 

DF 

Mean 

Square 

F 

Value 

Prob  >  F 

a  =  0.05 

Model 

6,024,287.18 

9 

669,365.24 

20.2233 

<  0.0001 

* 

T 

3,235,042.05 

1 

3,235,042.05 

97.739 

<  0.0001 

* 

Qt 

659,344.82 

1 

659,344.82 

19.9205 

<  0.0001 

* 

Ql 

794,562.33 

1 

794,562.33 

24.0058 

<  0.0001 

* 

T2 

212,422.81 

1 

212,422.81 

6.4178 

0.0124 

* 

Qt" 

558,906.79 

1 

558,906.79 

16.886 

<  0.0001 

* 

Ql" 

356,637.37 

1 

356,637.37 

10.7749 

0.0013 

* 

1 - 

P 

17,652.64 

1 

17,652.64 

0.5333 

0.4664 

TQl 

47,410.91 

1 

47,410.91 

1 .4324 

0.2334 

QjQl 

199,200.94 

1 

199,200.94 

6.0184 

0.0154 

* 

Residual 

4,633,827.61 

140 

33,098.77 

Lack  of  Fit 

660,829.65 

5 

132,165.93 

4.4909 

0.0008 

* 

Pure  Error 

3,972,997.96 

135 

29,429.61 

Cor  Total 

10,658,114.79 

149 
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Fitted  Model  -  Total  Kills 


co  = 


1595.89  +  179.867  +  81.200^+89.140^  + 

+  90.897"  +147.430^  -117.770" 

- 14.8570,  -  24.3470,  -  49.900,0,  ^ 


Lockheed  Martin 
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Perturbation  Plot 


>  Single  Factor 
Analysis 

-  Shows  curvature 
for  each  factor  at 
the  Center  point 

-  Provides  visual 
confirmation  of 
the  AN  OVA 
statistics 

-  “Opposing”  shift 
for  Quality  (C) 
reflects  value  of 
the  squared  term 
in  the  fitted 
equation 


DESIGN-EXPERT  Plot 

Total  Kills  -  Red 

Actual  Factors 
A:  Timeliness  =  0.00 
B:  Quantity  =  0.00 
C:  Quality  =  0.00 


2422.6152 


2076.7360 


=  1730.8568 


1384.9777 


1039.0985. 


Perturbation 


-1.000  -0.500  0.000  0.500  1.000 


Deviation  from  Reference  Point 


Cloned  View 


DESION-EXPERT  Plot 
Total  Kills-  Red 


2076.7K0  - 


Single  Factor  -  Quantity 


DoE  Tuturhj! 


-lal  x| 


One  Factor  Plot 


B:  Quantity 


Gauges  • 


Sheet  □! 


X  B:Quantity 

CiQuality  p 


1.00 


Default 


Term  B 


Zloned  View 


^xj 


DESION-EXPERT  Plot 
Total  Kills  -  Red  2422;&is2 

X=  B:  Quantit)^ 


One  Factor  Plot 


Design  Points 


aOTS.TWO  - 


Actual  Factors  "o 

A:  Timeliness®  1.00  (xi 
C:  Quality  =  1.00 


0:  Quantity 


Single  Factor  -  Quality 


DoB  Tutor  in! 


I  Cloned  View 


^Xj 


View 


DESION-EXPERT  Plot 


One  Factor  Plot 
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Conclusions 


•  The  model  is: 

-  Sensitive  to  the  C3I  parameters  of  Timeliness,  Quantity,  and 
Quality  of  Information 

•  The  Face-Centered  CCD  is: 


-  Statistically  Powerful 

-  Robust 

-  Capable  of  providing  significant  insights 
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